Data Noising as Smoothing in Neural Network Language Models
نویسندگان
چکیده
Data noising is an effective technique for regularizing neural network models. While noising is widely adopted in application domains such as vision and speech, commonly used noising primitives have not been developed for discrete sequencelevel settings such as language modeling. In this paper, we derive a connection between input noising in neural network language models and smoothing in ngram models. Using this connection, we draw upon ideas from smoothing to develop effective noising schemes. We demonstrate performance gains when applying the proposed schemes to language modeling and machine translation. Finally, we provide empirical analysis validating the relationship between noising and smoothing.
منابع مشابه
Investigating Bidirectional Recurrent Neural Network Language Models for Speech Recognition
Recurrent neural network language models (RNNLMs) are powerful language modeling techniques. Significant performance improvements have been reported in a range of tasks including speech recognition compared to n-gram language models. Conventional n-gram and neural network language models are trained to predict the probability of the next word given its preceding context history. In contrast, bi...
متن کاملEstimation of Reference Evapotranspiration Using Artificial Neural Network Models and the Hybrid Wavelet Neural Network
Estimation of evapotranspiration is essential for planning, designing and managing irrigation and drainage schemes, as well as water resources management. In this research, artificial neural networks, neural network wavelet model, multivariate regression and Hargreaves' empirical method were used to estimate reference evapotranspiration in order to determine the best model in terms of efficienc...
متن کاملContinuous space language models
This paper describes the use of a neural network language model for large vocabulary continuous speech recognition. The underlying idea of this approach is to attack the data sparseness problem by performing the language model probability estimation in a continuous space. Highly efficient learning algorithms are described that enable the use of training corpora of several hundred million words....
متن کاملA Comparison Between Time Series, Exponential Smoothing, and Neural Network Methods To Forecast GDP of Iran
متن کامل
Comparison of Three Decision-Making Models in Differentiating Five Types of Heart Disease: A Case Study in Ghaem Sub-Specialty Hospital
Introduction: cardiovascular diseases are becoming the main cause of mortality and morbidity in most countries. This research goal was to predict the types of heart diseases for more accurate diagnosis by data mining and neural network technics. Method: This research was an applied-survey study and after data preprocessing, three approaches of neural network, decision making tree and Bayes simp...
متن کاملذخیره در منابع من
با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید
برای دانلود متن کامل این مقاله و بیش از 32 میلیون مقاله دیگر ابتدا ثبت نام کنید
ثبت ناماگر عضو سایت هستید لطفا وارد حساب کاربری خود شوید
ورودعنوان ژورنال:
- CoRR
دوره abs/1703.02573 شماره
صفحات -
تاریخ انتشار 2017